CHX Network Issues Compromise Dual Data Center Connectivity Response

Situation

Impacts

Response

CHX trading applications lose their ability to communicate across data centers.

 

During Trading Hours:

 

Depending on the scope of the problem:

 

Monitoring of applications may be lost or compromised.

 

CHX may not be able to fulfill order processing and/or reporting obligations to National Market System.

 

Database loading in affected data center may no longer be up to date, and clerical support of order research and/or trade corrections in the affected data center would be impossible.

 

Post Trade Processing using the affected data center may be invalid.

 

All clerical and administrative operations would need to rely on the working data center’s database, including all Post Trade Processing.

 

1)      Confirm scope of impacts.

Are problems specific to:

-          SIAC and/or NASDAQ

-          CH2 and/or NY4

-          System Monitoring?

-          Matching Engines?

-          Inbound Market Data Processing?

-          Outbound Market Data Processing?

-          Firm Connectivity?

2)      Notify management.

3)      For each impact identified, see the appropriate Generalized Recovery Scenario in this documentation for appropriate responses, with a focus on whether or not CHX should be zeroing quotes and/or suspending trading.

System monitoring is a priority.

4)      Consider moving processing from the affected data center to the healthy data center.

See CHX Generalized Event - CHX Data Center Move Procedures.docx for more details.

5)      At time of resolution, see the appropriate Generalized Recovery Scenario in this documentation for appropriate integrity checks, and post trading impacts.

Scope of Impact Response Considerations:

1)      The network design is such that auto-failovers to redundant services should help the auto-recovery of any situation within 5 minutes of the event. 

If the problems are such that we don’t believe auto-failovers are working as expected, or that the problems may reoccur to the extent that our trading integrity is compromised, we should consider moving processing from the most adversely affected data center to the most healthy data center. 

See each application’s specific recovery procedures in this document for appropriate responses and considerations.